There Is a Digital Art History

In this paper, we revisit Johanna Drucker's question, "Is there a digital art history?" -- posed exactly a decade ago -- in the light of the emergence of large-scale, transformer-based vision models. While more traditional types of neural networks have long been part of digital art history, and digital humanities projects have recently begun to use transformer models, their epistemic implications and methodological affordances have not yet been systematically analyzed. We focus our analysis on two main aspects that, together, seem to suggest a coming paradigm shift towards a "digital" art history in Drucker's sense. On the one hand, the visual-cultural repertoire newly encoded in large-scale vision models has an outsized effect on digital art history. The inclusion of significant numbers of non-photographic images allows for the extraction and automation of different forms of visual logics. Large-scale vision models have "seen" large parts of the Western visual canon mediated by Net visual culture, and they continuously solidify and concretize this canon through their already widespread application in all aspects of digital life. On the other hand, based on two technical case studies of utilizing a contemporary large-scale visual model to investigate basic questions from the fields of art history and urbanism, we suggest that such systems require a new critical methodology that takes into account the epistemic entanglement of a model and its applications. This new methodology reads its corpora through a neural model's training data, and vice versa: the visual ideologies of research datasets and training datasets become entangled

A Computational Approach to Hand Pose Recognition in Early Modern Paintings

Hands represent an important aspect of pictorial narration but have rarely been addressed as an object of study in art history and digital humanities. Although hand gestures play a significant role in conveying emotions, narratives, and cultural symbolism in the context of visual art, a comprehensive terminology for the classification of depicted hand poses is still lacking. In this article, we present the process of creating a new annotated dataset of pictorial hand poses. The dataset is based on a collection of European early modern paintings, from which hands are extracted using human pose estimation (HPE) methods. The hand images are then manually annotated based on art historical categorization schemes. From this categorization, we introduce a new classification task and perform a series of experiments using different types of features, including our newly introduced 2D hand keypoint features, as well as existing neural network-based features. This classification task represents a new and complex challenge due to the subtle and contextually dependent differences between depicted hands. The presented computational approach to hand pose recognition in paintings represents an initial attempt to tackle this challenge, which could potentially advance the use of HPE methods on paintings, as well as foster new research on the understanding of hand gestures in art

Pose and Pathosformel in Aby Warburg’s Bilderatlas

look at Aby Warburg’s concept of Pathosformel, the repeatable formula for the expression of emotion, through the depiction of human pose in art. Using crowdsourcing, we annotate 2D human pose in one-third of the panels of Warburg’s atlas of art, and perform some exploratory data analysis. Concentrating only on the relative angles of limbs, we find meaningful clusters of related poses, explore the structure using a hierarchical model, and describe a novel method for visualising salient characteristics of the cluster. We find characteristic pose-clusters which correspond to Pathosformeln, and investigate their historical distribution; at the same time, we find morphologically similar poses can represent wildly different emotions. We hypothesise that this ambiguity comes from the static nature of our encoding, and conclude with some remarks about static and dynamic representations of human pose in art

A facial affect mapping engine

Facial expressions play a crucial role in human interaction. Interactive digital games can help teaching people to both express and recognise them. Such interactive games can benefit from the ability to alter user expressions dynamically and in real-time. In this demonstration, we present the Facial Affect Mapping Engine (FAME), a framework for mapping and manipulating facial expressions across images and video streams. Our system is fully automatic runs in real-time and does not require any specialist hardware. FAME presents new possibilities for the designers of intelligent interactive digital games

The perception of potential: interference, dimensionality and knowledge

This paper presents a system that investigates the sonification of wave interaction in a performance space and its interaction with a live performer – the illumination of sonic activity within a real space, in contrast to conventional ALife algorithmic, event- or material-based approaches. The model maintains three parallel representations of the entire live/virtual system: wavespace, symbol space and performance space. The cross-modal analysis and representation of behavior is important to the evolution of the system, which displays emergence on multiple levels of structure. Micro-evolution takes place within the population of wave-emitting and –listening agents. A higher level of structure emerges from their aggregate in interaction with the live performer, and a formal level as symbol space learns from the performer. Cross-modal representation is seen as a significant factor in the evolution of Western art music, in the development of multi-leveled structure and of work that affords many dimensions of engagement. We discuss the nature of knowledge produced through working with such systems and the role of the subject in ALife-generated knowledge. New models of simulation-derived knowledge are seen as important to cultural understanding

Painting by Numbers: Computational Methods and the History of Art

This thesis presents a new methodology for computational art history, in the tradition of Distant Reading from literary criticism. This method is based on operationalisation, the transcription of a concept or theory from cultural history into an algorithm. The method is explored through 3 extensive case studies, operationalising concepts from two major art historians of the twentieth century, Aby Warburg (1866-1929) and Michael Baxandall (1933-2008), and from the sixteenth-century Italian painter and art theorist Giovanni Paolo Lomazzo (1538-1592). Through quantitative analysis of visual phenomena, this thesis opens up the possibility of a new scale of art history through the use of computer vision. I conclude with some considerations on the kinds of art-historical thought amenable to operationalisation and computation

Early Modern Computer Vision

Abstract of paper 0820 presented at the Digital Humanities Conference 2019 (DH2019), Utrecht , the Netherlands 9-12 July, 2019

Totentanz : Operationalizing Aby Warburg’s 'Pathosformeln'

The object of this study is one of the most ambitious projects of twentieth-century art history: Aby Warburg's 'Atlas Mnemosyne', conceived in the summer of 1926 – when the first mention of a 'Bilderatlas', or "atlas of images", occurs in his journal – and truncated three years later, unfinished, by his sudden death in October 1929. Mnemosyne consisted in a series of large black panels, about 170x140 cm., on which were attached black-and-white photographs of paintings, sculptures, book pages, stamps, newspaper clippings, tarot cards, coins, and other types of images. Warburg kept changing the order of the panels and the position of the images until the very end, and three main versions of the Atlas have been recorded: one from 1928 (the "1-43 version", with 682 images); one from the early months of 1929, with 71 panels and 1050 images; and the one Warburg was working on at the time of his death, also known as the "1-79 version", with 63 panels and 971 images (which is the one we will examine). But Warburg was planning to have more panels – possibly many more – and there is no doubt that Mnemosyne is a dramatically unfinished and controversial object of study

Early Modern Computer Vision

Abstract of paper 0820 presented at the Digital Humanities Conference 2019 (DH2019), Utrecht , the Netherlands 9-12 July, 2019